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Abstract 

Phase clocks are synchronization tools that implement a form of logical time in distributed sys- 
tems. For systems tolerating transient faults by self-repair of damaged data, phase clocks can 
enable reasoning about the progress of distributed repair procedures. This paper presents a phase 
clock algorithm suited to the model of transient memory faults in asynchronous systems with 
read/write registers. The algorithm is self-stabilizing and guarantees accuracy of phase clocks 
within O(k) time following an initial state that is fc-faulty. 

Index Terms: distributed algorithms, fault tolerance, fault containment, synchronizers, self sta- 
bilization, time adaptive 



1 Introduction 

Measuring time is widely recognized as an important system service and greatly simplifies the construc- 
tion of many distributed algorithms. The reason, simply put, is that deductions about the progress of 
concurrent activities, made by measuring elapsed time, effectively substitute for communication and 
protocols that directly monitor such progress. Of course this technique can only be used to the extent 
that a distributed system is synchronous, matching its progress with elapsed time. Yet so attractive 
is the use of time to simplify algorithm construction, that even in asynchronous systems, researchers 
seek to simulate synchrony Q], introduce logical clocks |L6]], and/or logical time |l7j] as programming 
tools. 

One illustration of logical time in an asynchronous system is the organization of a computation into 
phases. The basic property of a phased computation is that a process does not enter phase (k + 1) 
until each related process has completed phase k. The case where all processes are related is equivalent 
to barrier synchronization, and the case where the relation between processes is specified by a graph 
corresponds to a phase clock. Many implementations of phased computation simply use a counter, 
called a clock, to represent the current phase number of a process. Consider the graph relation 
between processes to be a network communication topology, where the graph has diameter T> and 
distance between processes p and q is denoted by dist pq . A phase clock invariantly relates phase 
numbers and distance as follows: any process p has clock p = k + d only if clock q > k holds for each 
process q satisfying distpq = d (notice that k = 1 is just the basic property mentioned above). Thus 
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if clockp = k + d, holds at some state, we deduce that clock q — k holds currently or held at some 
previous state. This is a useful timing property because programs can use phase clocks for inferences 
about nonlocal information relayed through neighboring processes. For example, process p could use 
its clock to infer termination of a broadcast operation, rather than use explicit termination detection, 
by waiting for sufficiently many increments to clockp (assuming that the broadcast operation is geared 
to the phase clock). 

Phased computation is a reasonable discipline for many activities of a distributed system, including 
procedures invoked as part of fault diagnosis and repair. The fault domain for this paper is the 
model of transient faults, which corrupt local process states and communication registers, but do not 
damage a system's control logic. It is therefore feasible for a system to self-diagnose and restore 
variables corrupted by a transient fault to values that enable correct system function. One of the 
difficulties in using phase clocks to control distributed repair activities is that faults may corrupt local 
clock values. The phase clock protocol presented in this paper not only repairs clock values corrupted 
by a transient fault, but does so in a manner that enables the system to use the phase clock for other 
repair activities. 



Contributions. This paper presents a distributed phase clock, called the repair timer, specialized 
for the task of transient fault repair in a distributed system. The repair timer is time adaptive, meaning 
that it satisfies desired accuracy and progress properties within 0{k) time after any transient fault 
event corrupting k processes. The repair timer differs from standard phase clocks because it starts 
at zero and halts after repair is complete (behaving somewhat like an egg-timer); this enables direct 
inspection of elapsed repair time, which standard phase clocks do not provide]^]. The repair timer is also 
a self-stabilizing algorithm, able to restore all variables to a legitimate state following any transient 
fault event or combination of transient failures. Finally, the paper presents composition theorems to 
show how the repair timer is useful for the timing of fault repair procedures in a distributed system. 



Related work. Many recent works are motivated by what is seen as pessimism in the model of self- 
stabilization, which does not discriminate between cases of severe transient faults and minor transient 
faults. In addition to the desired robustness of self-stabilization, fast stabilization of output variables 
has been recently demonstrated in a number of algorithms JTl| , |, |5| and some general methods to 
achieve time adaptivity ||] or local self-stabilization ]10|, [l ] . 

Self-stabilizing phase clocks are given in ||, ^. None of these constructions guarantee fast sta- 
bilization for cases of limited transient faults, and all appear to require lengthy stabilization time 
(proportional to the diameter of the communication graph) in some cases where only a single process 
variable is corrupted by a transient fault. Requirements for a repair timer are described in which 
is a precursor to this paper. 



Contents. Section g presents the computation and system model for the paper. Section |3| presents 
the algorithm for the repair timer and Section ^ verifies the self-stabilization and time adaptive 
properties of the algorithm. To illustrate the use of the repair timer, Section ^| describes two designs 
incorporating the repair timer as a component in a system. The paper's concluding remarks are the 
subject of Section ^. Proofs of technical lemmas have been moved to the paper's Appendix. 

1 To see why this is not trivial, suppose some time-adaptive phase clock were available, and consider measuring 
elapsed repair time by recording the start time of repair in some local variable; such a local variable could, however, 
have an erroneous value due to a transient fault. Since transient faults do not provide any signal at the start of repair, 
a process cannot locally decide whether its local variables are accurate or not. 
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2 Distributed System 



The system consists of a fixed set of n processes that communicate by reading and writing shared 
registers. Communication between processes is limited to a network represented by an undirected, 
connected graph: for any pair of processes (p, q) there exist a pair (Register p<J , Register gp ) if and only if 
there is an edge between p and q in the communication graph. Process p is the only writer of Register p(? 
and q is the only reader of Register pg . A process cannot read the registers it writes. Registers thus 
approximate message passing with bounded buffers, and a self-stabilizing simulation of link registers 
using messages is described in & . A register can have numerous fields used to write values of different 
local variables (just as numerous local values can be transmitted in fields of one message). 

If (p, q) is an edge in the communication graph, then p and q are called neighbors, which is denoted by 
p G Afq or equivalently, q £ M p . The diameter of the communication graph is T>. The distance between 
any pair (p, q) in the graph is denoted by dist pq . The term region refers to a connected component of 
the graph that has some property of interest. 

Each process is an autonomous, finite-state computing entity. We use conventional imperative pro- 
gramming notation and concepts to describe the operation of a process, so each process has a program 
counter and program statements that manipulate variables. A subset of these variables are called out- 
put variables, which directly support the system's intended function. 

A configuration of p is a specification of values, one for each of process p's variables, the value of p's 
program counter, and a value for each register that p writes. A (system) state is a vector of process 
configurations, one configuration for each process in the system. Any function from the set of all 
states to the set {true, false} is called a state predicate. 

A process step is either a register operation (and corresponding advancement of the program counter) 
or some modification of internal and output variables (and program counter) of that process. A 
computation is an infinite sequence of states so that each consecutive pair of states corresponds to 
a process step and the sequence of states includes an infinite number of steps of each process. We 
thus assume that computations are fair; more precisely, we assume weak fairness in that no process is 
prevented from executing steps in a computation. We use the term computation segment to denote a 
finite, contiguous subsequence of a computation. 

The program of each process specifies a cycle, which consists of three parts: (i) a process reads the 
registers written by each of its neighbors, (it) the process possibly assigns values to its variables, and 
(Hi) the process writes registers for each of its neighbors. The definition of a cycle is a convenient and 
simple abstraction for measuring the progress of a process in a computation. 

The system is designed to accomplish some task represented by a state predicate Co- Whether or 
not Co holds at a given state is solely determined by the values of output variables. A predicate C is 
called a legitimacy predicate iff £ is a system invariant and C => Co- A state a is output-legitimate if 
Co holds at a, and is legitimate if C holds at a. It is often preferable to specify legitimacy (or output 
legitimacy) in terms of the behavior of processes rather than explicitly specifying a state predicate. 
A formal definition of legitimacy in terms of behaviors is possible, but to streamline the presentation, 
the state-based definition is used in this paper. Where process behavior is important in this paper, 
we verify separately that the system exhibits the desired behavior. 

Because each iteration of a process program specifies a cycle, time is conveniently measured in asyn- 
chronous rounds, which are defined inductively. A round of a computation, with respect to an initial 
state a, is a computation segment originating with a of minimum length containing at least one 
complete cycle (from reading registers to writing registers) of each process. The first round of a com- 
putation consists of a round with respect to the initial state of the computation, and round k of a 
computation, k > 1, consists of a round with respect to the first state following round k — 1. 
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A round is, roughly speaking, one unit of "parallel time" in the system. A notion similar to a round is 
commonly used to analyze the complexity of message-passing protocols by normalizing message delay 
to the maximum message delay ||. For analysis in this paper, the notion of a round is further refined. 
An Rp-round starting from a state a is a computation segment of minimal length containing at least 
one complete cycle of each process in the set { g | dist pq < d }. A round is thus equivalent to an 
R^-round for any choice of p. 

A system is self- stabilizing if every computation contains a legitimate state (that is, for any initial 
state, the system eventually reaches a legitimate state). The stabilization time is the worst case 
number of rounds in the prefix of a computation that does not contain a legitimate state. Proving 
that a system is self-stabilizing entails demonstrating that a predicate C is invariant, implies Co, and 
that every computation contains some state satisfying C. 

A fault event is a non-computational operation that modifies variables, program counters, and/or 
registers. More formally, a fault event can be any pair of states (whereas a consecutive pair of states in 
a computation is a process step). Computations do not include fault events; a system history could be 
a sequence of states consisting of computation segments punctuated by fault events. Reasoning about 
fault repair proceeds with respect to each computation segment, since the system cannot anticipate 
whether or not another fault will occur. 

A state a is k- faulty if fc is the minimum number of process configurations in a that, if appropriately 
changed, transform a into a legitimate state. The number k thus corresponds to the Hamming distance 
from a to the nearest legitimate state. There may be numerous ways to transform a to a legitimate 
state by changing k process configurations, some in which process p's configuration changes, and others 
where the transformation does not change p's configuration. It is convenient to resolve this ambiguity 
by some unique, deterministic choice of which processes should change configurations to obtain a 
legitimate state from fc-faulty state a. With such a deterministic choice, process configurations of a 
can be labelled faulty or nonfaulty depending on whether they should change or not. This deterministic 
choice can further be refined to label variables and register fields as either faulty or nonfaulty. How 
such a deterministic choice should be implemented turns out not to be an issue in the sequel; for the 
repair timer given in Section || there is an unambiguous definition of a faulty process configuration 
and for the interface proposed in Section [5] it is only required that if a faulty process configuration 
neighbors a nonfaulty process configuration, then the presence of a fault can be detected (which for 
many systems is the case even by reversing the choice of which of these two neighboring configurations 
is considered to be faulty). 

The main emphasis of this paper is time-adaptive, stable repair of output variables, meaning that a 
system should stabilize its output variables to satisfy Co from any fc-faulty initial state after at most 
0(k) rounds. Formally, a system is time adaptive if each computation starting from any fc-faulty initial 
state a contains an output-legitimate state a', within O(k) rounds following ex, such that every state 
following a' in the computation is output-legitimate. Given this emphasis, it is convenient to extend 
the terminology for faults: a process p is faulty (nonfaulty) in a computation iff p's configuration is 
faulty (nonfaulty) at the initial state. 

3 Algorithm 

One of the difficulties in using phase clocks to control distributed repair activities is that faults may 
corrupt local clock values. Indeed, repair of the clocks values is a primary concern of this paper, 
and the usual timing properties of phase clocks must be modified to cope with faults. Two goals for 
such modifications are: (a) clock values of processes not affected by faults can reliably be used for 
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inferences about nonlocal information; 
scope of the fault. 



(&) the response effort of the system is proportional to the 



Goal (a) seems relatively simple to satisfy, since the clocks of nonfaulty processes have predictable 
values. However for a standard phase clock, there are ambiguous cases of faulty situations. Suppose 
neighboring clocks have values x and x — 2 and only one of these two is a faulty value; there is no 
obvious way of distinguishing which of these two is faulty. The approach taken in this paper is to use 
a specialized phase clock for fault repair called a timer. Whereas phase clocks advance throughout 
system computation, the timer stops advancing when repair is complete. Thus each timer clock reaches 
a prescribed value T when the system state is fully repaired. If neighboring clocks have values T and 
T — 2, then we may conclude that the value T — 2 is due to a fault. 
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51 for (qeAfp) (x[q],y[q],r[q\,s[q}) <- read(Register gp ) 

52 if (wEcho V wMin < w) then w <— 1 + min(w, wMin, 3£>) fi 
if 

(cEcho A clock < T A -igap A wBigr A clock < cMin) 

then clock «— clock + 1 
(clock > T — P A gap) then clock. w«— 0, 
(clock < T — T> A gap A -^wBig) then clock <— w 
(cEcho A clock < T — T> A gap A wBigr A clock < cMin) 
then clock <— clock + 1 

fi 

S7 for (q G M p ) write( Register p(J 



(clock, w,x[g],y[g]) ) 



Figure 1: timer for process p 



Variable Conventions. The variables appearing in Figure [j] are local variables of process p. A 
number of proof arguments are statements relating variables of different processes, and subscripts are 
used to distinguish variable ownership (for instance, clock g is owned by q). Similarly, the predicates 
defined in Figure [I] are subscripted in definitions and proof arguments (such as gap p for process p) . 

Statement SI copies four register fields to four local variables, (x,y,r,s). Call these variables the image 
variables. Implicitly the code of Figure [l] defines a mapping from each image variable to a register 
field and a corresponding "base" variable of a neighboring process (written by statement S7). We say 
that each image variable is based on a variable of a neighboring process, meaning that the value of 
an image variable is copied (via register communication) from the variable upon which it is based. 
Variable x p [q], for example, is based on clock g . Register fields are also images that are statically based 
on variables. 

The meaning of time adaptivity described in Section || depends on declaring some of the process 
variables to be output variables. For the repair timer, let clock p be the output variable of process p. 
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The output correctness for clock variables is the subject of Section 



4.2 



Program Conventions. The statements S1-S7 given in Figure |lj describe one complete cycle of 
the repair timer for process p. Therefore, after executing S7, process p executes SI to start the next 
cycle. The group of statements S3-S6 constitute a multiway if statement; in any cycle, at most one 
of S3-S6 are executed. 

Statements S2-S6 specify internal calculations for process p, since they manipulate local variables. In 
a computation, we suppose that each of these statements specifies one computation step. Statements 
SI and S7 specify each |AC,| computation steps, since a step can read or write at most one register. 
Ordering of the read and write operations of SI and S7 is unimportant to the algorithm. 

Algorithm Structure. To understand the algorithm of Figure |l| it is useful to first ignore statements 
S3-S6 and focus attention on the w variables. Notice that statement S2 will reduce w p if any of the 
registers read by SI imply a value w p — 2 or smaller for any neighboring w variable. The global 
effect of many processes executing S2 can thus be "convergence to the least w" over a number of 
rounds. The result of executions of S2 will, in general, lead to a situation where neighboring process 
w variables differ by at most 1, which is one of the properties of a phase clock. The wEcho condition 
of S2 allows any process with a globally minimal w variable to increment its w variable after all 
neighbors acknowledge its current value, via the s image variables (which occurs within two rounds). 
Therefore the set of w variables apparently enjoy both properties of a phase clock — that neighboring 
w variables differ by at most 1 and increase continually (until the upper bound of 32? + 1 occurs) in 
a computation. 

Why not simply use the w variables for repair timing and dispense with the logic of S3-S6? The 
answer lies in the additional constraint we impose for faulty initial states. For repair purposes, it is 
not enough for clocks to be in phase and increment, they should also be accurate, meaning that the 
value of a clock should be a measure of how long computation has progressed after the initial detection 
of a fault. The w variables do not have this property. For instance, a faulty initial state could have 
2? as the initially smallest w variable, so that all subsequent states have w variables overstating the 
repair time at least by 2?. An attempt to fix this problem would be some statement similar to S4, that 
would reset w to zero whenever neighboring w variables differ by more than 1. It is easy to construct 
examples of computations where such an attempted fix will fail because w variables are reset to zero 
infinitely often. This kind of idea can work, however, if any w variable were guaranteed to be reset 
to zero at most once in a computation, and that is the basic idea behind statements S3-S6, which 
reset a clock variable to zero at most once in a computation. Although w variables do not enjoy the 
accuracy needed for repair timing, they provide a useful "reset layer" for the clock adjustments of the 
algorithm. 

Definition 1 A state is timer-final if every register field and image variable value is equal to the 
value of its base variable, and (Vp :: clock p = T A w p = 32? +1). A process configuration is 
timer-final if all its variables and register fields have values corresponding to a timer-final state. We 
define predicate Ct to hold for a state iff that state is timer-final. 

The value T used in the algorithm and Definition [l] is a constant adequate for the fault tolerance 
of the repair timer and for the application of the timer, as discussed in Section ^[ The proof of 
self-stabilization of the repair timer requires only that T > 412?. 
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Verification. The verification of desired repair timer properties is divided into two stages. First, 
the algorithm is considere d as an isolated component, so that faulty states are those states deviating 
from Definition Section 4.1 is devoted to a proof that the repair timer self-stabilizes to a timer-final 
state. Section 4.2 presents the proofs that apply to /c-faulty initial states, showing that the repair 
timer achieves desired accuracy after 0(k) time following the initial state. 

The second stage of verification is concerned with integration of the repair timer as a component of a 
system. The timer is a tool for time adaptive repair. Discussion of how the timer is used is deferred 
to Section ^|, where it is explained that the timer is a service with only one operation, namely to start 
the timer by assigning clock <— 0; thereafter, the clock should increment as a phase clock. Although a 
system state's legitimacy depends on variables of all system components, the simple interface between 
the timer and other system components makes it reasonable to consider fault tolerance properties of 
the timer in isolation, which motivates the two stage approach to verification. 



4 Stabilization and Adaptivity 
4.1 Self-Stabilization 

Each process writes its communication registers in every cycle from its variables. Therefore, following 
the first round of any computation, all register fields are equal to current or previous values of the 
corresponding base variables. Following the second round, each image variable has a value previously 
written from the corresponding base variable. Moreover, following the third round of a computation, 
the third and fourth fields of Register^ contain values previously written by p and then copied by 
q. It is convenient to assume that register fields correspond to values previously written in the 
computation, so we call a computation based if it is the suffix, starting from round three or higher, of 
another computation. 

Statements S4 and S5 have the only assignments that may reduce the value of clock variables. We call a 
computation (or computation segment) reset-free if no process executes S4 or S5 in that computation. 
A computation is called rising if it is the suffix of a based, reset-free computation such that each process 
has read its registers at least once in the based, reset-free computation prior to the first state of the 
suffix. Rising computations enjoy the useful property that at all states, the value contained in x p [q] is 
a lower bound on the current value of clock g . (This property follows because the computation is reset- 
free and each process previously read registers and assigned to its x variables while the computation 
was reset- free.) 

Definition 2 

b pq = (p (ji Afq) V (|clock p — clock 9 | < 2 A Xp[q] < clock 9 A |clock p — x p [q] \ < 2) 

A state is smooth if (Vp, q :: b pq ). A set of processes P forms a smooth region if the subgraph of the 
communication topology induced by P is connected and (Vp, q : p, q G P : b pq ). ■ 



Lemma 1 In a rising computation, (b pq A b qp ) is an invariant for any pair of processes p and q. 



Lemma 2 Let a be the first state of a rising computation segment such that for p G Af q , both clock p 
and clock g have incremented at least once in the computation segment. Then (b pq A b qp ) holds at 
state a. 
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Lemma 3 If each clock variable has incremented at least once prior to state a in a rising computation 
segment, then a is smooth. 

Lemma 4 Smoothness is invariant for a based computation; within 0(2?) rounds following a smooth 
state, a based computation contains a timer-final state. 

Lemma 5 If clock p is less than T and less than or equal to all neighboring clock values at the initial 
state of a based, reset-free computation segment, and w p = 32? + 1 holds at the initial state, and 
this computation segment contains at least two rounds, then clock p increments at least once in the 
computation segment. 

Lemma 6 Let the initial state of a based computation satisfy (Vp :: clock p < 72? A w p — 32? + 1). 
The computation contains a state where (3q :: clock g = 102?+ 1); the first state satisfying (3c/ :: 
clock g = 102? + 1) is a smooth state. 

Lemma 7 Let the initial state of a based computation satisfy (Vp :: clock p < 72? A w p = 32? + 1). 
Within 0(2?) rounds, the computation contains a smooth state. 

Lemma 8 Consider a based computation such that (clock r = A w r = 0) holds for some process r 
in the initial state. Within 2? rounds there is a state satisfying (Vp :: clock p < 32? A w p < 32?). 

Lemma 9 Consider a based computation such that (clock,. = A w r = 0) holds for some process 
r in the initial state. Within 0(2?) rounds there is a smooth state or there is a state satisfying 
(Vp :: clock p < 72? A w p = 32? + 1). 

Theorem 1 The timer stabilizes to a timer- final state (satisfying Ct) in 0(2?) rounds. 

Proof: The invariance of Ct is verified by observing that none of S1-S6 change any variable value 
at a timer-final state. Convergence is demonstrated by a sequence of claims about an arbitrary 
computation A. Let B be a suffix of A beginning following the second round of A; by definition, B is 
a based computation. We consider two cases for B. 

Case: B contains no step executing S4 within 0(2?) rounds. By arguments similar to those given 
in the proof of Lemma ^, some state of B satisfies (Vp :: w p = 32? + 1) within 0(2?) rounds and 
continues to hold at least until S4 executes. Let C be a suffix of B satisfying (Vp :: w p — 32? + 1) at 
its initial state. Observe that C is based and reset-free for 0(2?) rounds, so Lemma || is applicable to 
C. Within 0(2T) rounds of C, (Vp :: clock p = T) holds, and the state satisfies Ct- 

Case: B contains some step executing S4 within 0(1?) rounds. Execution of S4 results in a state 
satisfying the premise of Lemma ^[ Therefore B either contains a smooth state within 0(2?) rounds, 
or contains a state satisfying (Vp :: clock p < 72? A w p = 32? + 1) within 0(2?) rounds. The latter 
possibility is the premise for Lemma [7| which shows that a smooth state is subsequently obtained 
within an additional 0(2?) rounds, so with either possibility, B contains a smooth state within 0(2?) 
rounds. Lemma |^ implies that B contains a timer-final state within 0(2?) rounds following a smooth 
state. ■ 
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4.2 Time Adaptivity 



The desired fault tolerance of the timer consists, informally, of the following two properties. (1) 
Within fc rounds from a fc-faulty initial state, every clock is accurate, that is, if clock p = t for t < T, 
it should be that p has incremented clock p as a phase clock t times during the repair procedure. (2) 
Each faulty process clock is reset to zero and subsequently increments as a phase clock, incrementing 
to fc within O(fc) rounds. 

Property (1) provides the accuracy needed so that a process can safely wait for distant information 
to be reliable. Property (2) assures that such distant information arrives in a timely fashion. Because 
faults may damage clock and other timer variables, Theorem ^ below provides a conditional form of 
(1), necessarily relaxed to accommodate unusual initial states. Also, some unusual cases of initial 
states require a conditional form of (2), provided by Theorem ||. 

A system state is faulty if it does not satisfy the definition of legitimacy. In considering the timer 
in isolation, a state is fc-faulty if no fewer than k process configurations require change to obtain a 
timer-final state. However, a complete definition of system legitimacy depends on components other 
than the timer, so a limited notion of fault is appropriate for the timer. 

Definition 3 A set of processes P is unperturbed at state a if P forms a smooth region, (Vp : p € 
P : clockp > T -V), and (Vp, q : p G P A q S Af p A q g P : clock p = T A x p [q] > T-l). A process 
p is unperturbed at a if there exists an unperturbed region containing p; process p is perturbed if there 
exists no unperturbed region containing p. State a is k-perturbed iff k is the number of perturbed 
processes at a. ■ 

The motivation for this definition derives from the ambiguity of certain clock values and nondeter- 
minism of asynchronous computation. Some proofs are simplified using Definition ^, which defines a 
perturbed process to be a weakening of a faulty process configuration (a nonfaulty process configura- 
tion is unperturbed, but the converse may not hold). It follows that if the timer algorithm satisfies 
desired properties (l)-(2) within k rounds from any fc-perturbed state, then similar properties also 
hold for any fc-faulty initial state. Definition ^ is not useful if k = 0, so in the sequel any reference to 
fc-perturbed state is assumed to imply k > 0. 

Definition 4 Within a computation, a variable clock p is d-accurate at a state a if clock p > T — V 
holds, or if clock p <T — T> implies, for < m < T>, that the number of R™-rounds completed prior to 
state a is at least (clock p — m — d), and that for every process q, the value of clockp has incremented 
at least (clock p — dist pq — d) times prior to state a in the computation. For a computation initiating 
from a fc-perturbed state, a state a is time- accurate if for unperturbed p, clock p is <i-accurate for 
d = 2 • min(fc, £>), and for perturbed p, clock p is d-accuratc for d = 5 • min(fc, T>). ■ 

Definition [| falls short of the desired precision of property (1), but satisfies safety concerns for many 
situation of repair timing because a G?-accurate clock provides a lower bound on the number of cycles 
that distant processes have completed during repair. For instance, a repair application could depend 
on a distributed procedure that terminates after m clock increments in a non-faulty environment; this 
application could wait for d + m clock increments if the repair timer ensures only d-accurate clock 
variables. Unfortunately, an initially faulty state can have arbitrary values in faulty process clock 
variables, making it impossible to instantly have time accuracy. Theorem ^| given at the end of this 
section states that time accuracy is guaranteed from any fc-faulty initial state, provided fc < n, after 
at most min(fc,2?) rounds of computation. 

Lemma 10 Any process p executes S4, resetting clock p and w p , at most once in any computation. 
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Lemma 10 is a corollary of arguments given in the proofs of Lemmas || and ^|. It is useful to know 
that processes execute S4 at most once because any reset step subsequent to S4 is therefore due to 
S5. Arguments in the proof of Lemma || show that w values increase if S4 does not execute, and this 
idea can be used to establish the eventual increase of clock values. 



Lemma 11 Let a be a result of p executing S4. Then for any process q satisfying dist pq = t < 
mm(k,T>) there occurs a state a', within t rounds following a, such that clock, < 3t A w q < 3i; and 
if there is a path consisting of unperturbed processes from p to q, then a state a" occurs within t 
rounds following a such that clock 9 < t A w q < t. 

Lemma 11 considers a level of detail not discussed in the proof of Lemma |8[ which supposes based 
computations. Lemma [n] can also be extended to distances beyond k, shown in the following. 



Lemma 12 In any computation starting from a fc-perturbed initial state, for each unperturbed 
process p satisfying dist pq = t with respect to some perturbed process q, the following holds: process 
p executes S4 within 4 + min(P, k + 1) rounds. 



Lemma 13 In any computation beginning from a fc-perturbed state, any process p satisfying 
dist pq = t from some perturbed process q does not execute S4 after round 4 + min(22?, t + 2k). 



Lemma 14 Let a be a result of p executing S4. Then for any process q, within t rounds following 
a there occurs a state a' such that w q > min( [(t— dist pq )/2\ , 3D + 1) is invariant for the computation 
beginning with a'. 



Lemma 15 Let a be a result of p executing S4. Then for any process q, within t + 2 rounds 
following a there occurs a state satisfying clock g > min(L((f — 2) — dist pq )/2\ , T). 



Theorem 2 Any computation starting from a fc-perturbed initial state, k < n, contains a time- 
accurate state a after at most min(fc, T>) rounds following the initial state, and all states following a 
are time-accurate states. 

Proof: Provided k < n, arguments in the proof of Lemma ^| show that for each perturbed region 
R, some process r executes S4 within the first round, where r satisfies either r € R or r € Af q for 
some q G R. Lemma ^ then implies that within min(fc, T>) additional rounds, each p 6 R satisfies 
clockp < 3 • min(A;,P). Each unperturbed process clock variable remains larger than T — T> until S4 
is executed, which resets the clock to zero. Thus within min(/c, T>) rounds, each clock p is either larger 
than T — T> or is at most 3 • min(/c, T>). After min(fc, T>) rounds, unperturbed processes can decrease 
clock variables to zero, but such a decrease does not falsify the conditions for a time-accurate state. 
Therefore, to show time accuracy, it suffices to show that increments to clock p imply corresponding 
increments have executed at distant processes. 

After a process p executes S4, it does not increment clock p until cEcho holds. If an unperturbed q 
is a neighbor of p. then p does not increment clock p until q has reset clock 9 and updated the image 
variables and register fields so that p observes cEcho. It is a simple induction to show that clock p 
cannot increase to a value t unless q has incremented clockg at least (t — 1) times. Now consider a 
minimum length path P of processes, of length d, from p to some process r, such that each process in 
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P is unperturbed. By a double induction, on t and d, it follows that clockp cannot increase from zero 
to t unless each process q & P has incremented clock g at least t — dist pq times. The same argument 
shows that processes of P complete at least the same number of Rp-rounds in the period where clock p 
increases from zero to t. 

Returning to the event of p executing S4, we now consider the case of perturbed q € Af p . As observed 
in the proof of Lemma [ll], it is possible that p can increment clockp twice before q completes a cycle 
because corrupt values in the initial state enable the cEcho and wEcho conditions. Furthermore, p 
can increment clockp a third time before q increments its clock because q completes a cycle to enable 
cEcho p . However in the case of such a third successive increment by p, clockp > clock g and b pq A b qp 
hold as a consequence. Thereafter, we reason about the interaction between p and q as for unperturbed 
neighbors (note that any subsequent executions of S5 by p or q validate this argument, since we reason 
about the highest value attained for clock variables after p's initial three increments). Therefore, the 
value of clockp does not increase to t unless q has incremented clockp at least t — 3 times. Again, we 
may consider a minimum length path P of processes, of length d, from p to some process r, such that 
each process in P is perturbed (with the possible exception of p) . By a double induction, on t and 
d, it follows that clockp cannot increase from zero to t unless each process q € P has incremented 
clock g at least t — 3 • distpq times. Similar arguments show the completion of the appropriate number 
of Rp-rounds while clockp increases from zero to t. 

Notice that in the case of a perturbed path of processes, accuracy can diminish by two extra clock 
units per unit of distance, whereas in the case of an unperturbed path, accuracy corresponds precisely 
to distance. These observations combined can be used to verify that in any minimum length path P 
from p to r, after p executes S4, the value of clockp increases to t only if for each q G P, the value of 
clock g has incremented at least t — dist pq — 2m times, where m is the number of perturbed processes 
in the subpath of P from p to q. Since m < min(/c, T>) 1 time accuracy is verified for p. 

The arguments above show that time accuracy holds for all unperturbed processes within min(/c, T>) 
rounds and that any subsequent state is 2 • min(fe, "D)-accurate for unperturbed processes. For per- 
turbed processes, similar reasoning applies. Instead of relying on S4 to establish the baseline clock 
value, we use instead a value bound by the construction given in Lemma [H^s proof. Within min(fc, T>) 
rounds, there is a state a' where perturbed p has a clock value of at most 3j, and j < min(/c, T>) is the 
distance to some unperturbed process that executes S4 in the first round. The value of clockp cannot 
increase from 3j to 3j + t unless process q has incremented its clock at least t — dist pq — 2m times, 
where m is at most min(A;, T>). Therefore when clockp = x at some state following a', we infer that 
clock g has incremented at least x — 3 • min(fc,2?) — dist pq — 2 ■ min(fc,2?) times, which verifies time 
accuracy for unperturbed processes. ■ 

Theorem || addresses desired property (1) set out at the beginning of the section. Property (2) 
specifies that each faulty process clock be reset to zero and then advance as a phase clock. For the 
same reason that (1) has been weakened to the time accuracy of Definition ^, we weaken (2) to require 
only that each perturbed process be reset to some value in the range [0, 3 • min(/c, T>)\ within k rounds 
following the fc-faulty initial state, and thereafter increments as a phase clock. Theorem ^ implies 
that subsequent increases to clock values satisfy a distance property relating the value of a clock to 
the number of increments of other clock variables. The following theorem states the weakened form 
of (2). 

Theorem 3 For any computation starting from a /c-faulty initial state, k < n, each perturbed 
process clock is at most 3 • min(/c,P) within min(fc,2?) rounds and increases to value \_((t — 4) — 
min(fc, T>))/2\ within t rounds; and each unperturbed process clock similarly increases to \_(t — 4)/2j 
within t rounds after resetting by S4. 
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Proof: Lemma 11 directly shows that perturbed processes assign clock variables to at most 3 • 
mm(k,T>) within the first min(/c,2?) rounds. Lemma 15 establishes that p increases its clock to at 
least m = min(|_((i — 2) — dist pq )/2\ after t + 2 rounds following the execution of S4. Lemma |l2| 
establishes that for each perturbed region, some process p either within or neighboring the perturbed 
region executes S4 in the first round. Lemma [l5| establishes that processes within a given distance 
increase their clock values as clock p increases. Any process q within a perturbed region containing 
or neighboring p is at most distance min(/c, T>) from p; simplifying the bound of Lemma [l5] using 
min(fc, V) as a distance upper bound yields a lower bound of clock g > [((t — 2) — min(A;,X>))/2j after 
t + 2 rounds. ■ 



5 Embedded Timer 

This section discusses use of the repair timer as a component in a system. Whereas Section |4| investi- 
gated properties of the repair timer in isolation, the results of this section are essentially composition 
theorems stating conditions under which the repair timer can be used as a tool to enable time-adaptive 
fault tolerance in a system. 

Consider a system that uses the repair timer as one of its components. The term core system is used in 
this section to refer to all system components outside the repair timer; in other words, the entire system 
consists of the core system plus the repair timer. The elements of a process configuration (variables 
and registers) can be partitioned into those belonging to the repair timer and those belonging to the 
core system. The timer projection of a state is formed by removing all elements from each process 
configuration not relevant to the repair timer (that is, only clock, w, related image variables and 
register fields are retained). A core projection is formed by removing all repair timer elements from 
the state. 

Requirement 1 Output legitimacy Co of the system is defined solely in terms of the core projection, 
that is, no repair timer variable is an output variable. Core system legitimacy, given by the predicate 
Cc, is also defined with respect to the core projection; predicate Cc is independent of repair timer 
variables or register fields. The legitimacy predicate for the system is C = Cc A Ct- ■ 

The interface between core system and repair timer is illustrated in Figure |^. Communication between 
these two components occurs in each process, but is restricted to two methods: the core system can 
reset the clock and w variables, and the core system may read the current clock value. Henceforth the 
term double-reset is used to denote the assignment clock, w <— 0, 0. Both S4's assignment of Figure [j] 
and the core system's assignment illustrated in Figure are double-reset assignments. 



clock, w <— 0, 



core system 



repair timer 



read clock 



Figure 2: interface between core system and repair timer 
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So that results from Section |j are applicable to the composite system, each process invokes the repair 
timer (statements S2-S6) once in each cycle. Figure [l] includes SI and S7 to present the repair timer 
in isolation, however in the context of a system invoking the repair timer, these two statements would 
be subsumed by statements reading registers at the beginning of a process cycle and writing registers 
at the end of a cycle. 

A process configuration can be faulty with respect to the repair timer elements, the core system 
elements, or a combination of both elements. If a state cr's core projection violates Cq then ct is said 
to be core-faulty; if cr's timer projection violates Ct then a is timer-faulty. While Definition |l| provides 
the basis for a precise characterization of a faulty repair timer, the situation for a general system can 
be ambiguous, as observed in Section |2|. 

Requirement 2 If p's process configuration is not core-faulty and Register 9p is faulty at ct, then the 
presence of a fault at ct can be detected from the variables of p and the contents of Register^. ■ 

In many cases it is not difficult to design a system satisfying Requirement ||, in spite of the ambiguity 
of a faulty process configuration — the requirement only specifies that p detect the presence of a 
fault, and p is not required to determine the fault's location (fault identification remains ambiguous). 
Depending on the particular computation, p may not detect a fault. For instance, q may repair its 
configuration, changing the contents of Register p? , before p reads the register. 

The importance of Requirement ^ is that nonfaulty p has the capability to detect a fault, retain the 
current values of its output variables, and initiate repair procedures. Moreover, p can "contain" the 
fault because it reacts before copying values from Register^ and transmitting them to other processes. 

Requirement 3 Each cycle of a process invokes the repair timer. If, after reading registers at 
the start of a cycle, a fault can be inferred (as described in Requirement |||) for process p, and if 
(clock > T — T>), then p executes a double-reset. No other statements of the core system change the 
clock or w variables; any number of statements of the core system may read the clock variable. The 
legitimacy predicate for the core system does not depend on the clock or w variables of the repair 
timer. ■ 



Requirement 4 If any process p executes a double-reset resulting in a state ct, then within T — 71? 
rounds following ct, the core system component of the state is legitimate. ■ 

Requirement || means, for most core systems, that the core system stabilization time Ai satisfies 
Ai < T — IT). In essence, this is a constraint on T, which is added to the constraint T > 11X> given 
in Section ||. 

Lemma 16 If the core system is self-stabilizing with stabilization time Ai and satisfies Require- 
ments 0-^, then the system is self-stabilizing with stabilization time Ai + 0(T), and the double-reset 
assignment executes at most once for each process in any computation. 

The proof of Lemma [R3] rests on the independence of the core system and the repair timer, as specified 
by Requirement |3|, and the fact that the core system stabilizes before there is any possibility of 
executing a second double-reset by any process. The requirements do not, however, preclude the design 
of the core system from depending on repair timer properties. For instance, proving stabilization time 
Ai for the core system may depend on timer accuracy, since the core system can read clock variables 
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during convergence to Cc, and timer accuracy can be used in some circumstances to measure the 
progress of distributed algorithms and to allow processes to wait for such algorithms to stabilize. 

More interesting than using the repair timer for stabilization is the use of the repair timer to enable 
time adaptive repair of output variables. The remainder of this section illustrates the use of the repair 
timer in two designs. Design [l] is a time adaptive system, repairing output variables in 0(min(fc,X>)) 
rounds from any fc-faulty initial state. The design requires that the core system use a sequence of repair 
procedures, following an idea developed in Q . Output variables of nonfaulty processes may change to 
illegitimate values during convergence, but all output variables satisfy Co within 0(min(fc, V)) rounds 
and continue to satisfy Co thereafter. Design || is not fully time adaptive, but illustrates another use 
of the repair timer: the system can repair output variables in 0(r) rounds from any fc-faulty initial 
state, fc < r, and no nonfaulty process changes an output variable during repair. 

Design 1 The core system has T> independent repair procedures, denoted repair 1 for 1 < i < T>. 
Each of the repair procedures uses its own set of variables, including variables that are intended to 
be copied to the core system's output variables. Let output 1 denote the set of variables of repair 1 that 
correspond to the system's output variables, and let repair^ denote process p's portion of repair*. We 
suppose that the core system also prepares a set of variables output c intended to be copied to output 
variables. Each repair procedure is invoked in every process cycle and repair* is self-stabilizing to a 
predicate C l within M. rounds. When Cc holds, the output 1 variables are equal to the system's output 
variables, for 1 < i <T>, and output* 7 is also equal to the system's output. 

Procedure repair 1 has the property that, if the initial state is j-faulty, for j < i, then for all p, within 
h ■ i rounds, there occurs a state a such that for all p, the variables of output 1 satisfy Co (modulo 
renaming or copying their values to the system outputs) at a and all subsequent states. To exploit the 
repair timer, we suppose a stronger convergence property for repair 1 , namely that output^ variables 
stabilize within h ■ i of the R p -rounds. 

Given any fc-faulty initial state satisfying k <T>, certain repair procedures agree on values for output 
variables: for i > fc, after repair 1 stabilizes output 1 , any procedure repair^ for £ > i stabilizes output* 2 
to the same values that output 1 has. Moreover, the core system stabilizes output c to the same values 
contained in the stabilized output fc variables. The stabilized values of output sets are also constrained 
by distance from a fault: for any nonfaulty p such that the minimum distance from p to a faulty 
process is d, where d > fc, then all of the output p sets stabilize to the same values already contained 
in p's output variables. 

The output p sets are copied to the output variables of process p as follows. In each cycle, if clock p = T, 
then p copies output^ to its output variables. Otherwise, in each cycle, p copies output^ to its output 
variables where i is the largest value satisfying 1 < i < V and (h + 5) • i < clock p . No output p set is 
copied to the output variables if (h + 5) • V < clock p < T holds. ■ 

Theorem 4 If a system for Design |l| satisfies Requirements [l]-[| and M. = 0(T>), then within 
0(min(fc,2?)) rounds following any fc-faulty initial state, the system output-stabilizes to Co- 

Proof: Consider a fc-faulty initial state. If fc > T>, then from M = 0(V) and Theorem |J the system 
stabilizes to C and hence Co in 0(T>) rounds, which proves the conclusion. The remaining case is 
fc < T> for a fc-faulty initial state. For this case, we first show that any faulty process clock is time 
accurate within the first fc rounds. Requirement || ensures that some process within distance fc from 
any faulty process executes a double-reset in the first round, and Theorem || implies subsequent time 
accuracy within fc rounds. The same argument implies that each nonfaulty process within distance fc 
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from a faulty process has a time-accurate clock after at most k rounds. All nonfaulty processes have 
time-accurate clock variables throughout the computation. 

Design [l] specifies that some nonfaulty processes do not change their output variables by any repair 
procedure, so the proof obligation is to show that faulty processes and those nonfaulty processes 
within distance k to a faulty process stabilize their output variables in O(k) time. After k rounds, all 
such processes have time-accurate clock variables. By definition of time accuracy for a fc- faulty initial 
state, a time-accurate clock p variable with value t implies that the number of R p -rounds preceding in 
the computation is at least t — 5fc. Procedure repair fc converges within h ■ k of the R p -rounds, so after 
time accuracy holds, a clock p value of h ■ k + 5k = (h + S) ■ k implies that variables of output^ can be 
copied to p's output variables. The conditions of Design [l] also justify copying output^, to the output 
variables when clock p > (h + 5) • j for k < j < V. 

Having established the safety of copying output sets to the output variables, the remaining obligation 
is to show that all such copying either completes within 0(k) rounds or that any subsequent copying 
will not affect Co ■ Theorem || implies that all processes within distance A: to a faulty process will, 
after time accuracy holds, increase their clock variables to (h + 5) • k within 0(k) rounds and will 
not subsequently decrease their clock values below this value. Therefore, within 0(k) rounds, all 
processes within distance k to a faulty process assign their output variables, while those processes 
further than distance k from a fault do not assign their output variables to falsify Co by any step of 
the computation. ■ 

The repair* procedures of Design [j] are independent, meaning that they do not share any of the 
variables they modify. Because the variables of repair are inactive for nonfaulty processes during the 
period of stabilization, they are a resource for faulty processes: values from nonfaulty output^ can 
be disseminated to other processes and used for the stabilization of repair 1 procedures. For details on 
this technique, illustrated in a synchronous computation model, the reader is referred to 

Design 2 The core system uses procedure repair r with a set of variables denoted output^ that are 
equal to the system output variables at a legitimate state. Procedure repair 1- stabilizes the output 1 " 
variables to satisfy Co within h ■ r time from any initial state that is j'-faulty for j < r; each faulty 
process p stabilizes output^ after at most h ■ r of the R p -rounds occur. The output^ sets are copied to 
output variables of process p as follows. In each cycle, if clock p > (h + 5) • r, then p copies output p 
to its output variables; for all other values of clock p process p leaves its output variables unchanged. 
The repair r procedure stabilizes output p to values already contained in p's output variables for any 
nonfaulty p. ■ 



Theorem 5 If a system for Design g satisfies Requirements Q-g, then within 0(r) rounds following 
any k- faulty initial state for k < r, the system output-stabilizes to Co- No step modifies output 
variables of nonfaulty processes to values differing from those specified by Co- 

Theorem ^| can be verified by reasoning similar to the proof of Theorem |]. Design ^| is not self- 
stabilizing and Theorem || does not specify Requirement |J as a condition. The fault tolerance of this 
design is limited to r faulty processes. 



6 Concluding Remarks 

It is challenging to construct a system that can repair variables inflicted by transient faults. A 
reasonable methodology for such system construction is based on tools for fault detection and repair, 
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and these tools must themselves satisfy properties of time adaptivity and stabilization. This paper 
presented a phase clock algorithm specialized for the task of fault repair. The designs presented in 
Section ^ show how the repair timer can be composed with other system components. 

Although time adaptivity and self-stabilization are major themes for this paper, the repair timer can 
be useful even when neither full stabilization nor fast stabilization is needed, because it is convenient 
to reason about the progress of repair procedures by measuring elapsed time (which would otherwise 
be complicated due to possible corruption of time-measurement variables). An observer of the system 
located at process p could monitor repair progress by repeatedly examining clock p , possibly delaying 
critical activity until repair is complete. 

Use of the repair timer can add overhead to repair procedures because each cycle of repair invokes 
the timer, and the clock variable only increments in relation to rounds. It could be that actual repair 
only involves a small subset of processes, but a clock variable will not, in general, increment t times 
unless all processes at distance d have completed t — d cycles — including processes that are not 
involved in the repair. Thus the measurement of repair time in rounds could be overly pessimistic 
and cause processes to wait longer than necessary before they infer that repair is complete. Another 
slowing of repair timing results if a loose upper bound on the network diameter is used for T> (an 
upper bound is typically proposed for dynamic networks) since T, the "resting value" for the repair 
timer, is determined by the value T>. 
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7 Appendix: Proofs 

Proof of Lemma [j]: The essence of the proof is that neither S3 nor S6 increment a clock to a value 
two greater than any neighbor. Since Definition || involves x variables, the effect of statement SI 
requires examination. Reading a register to assign an x variable only increases the accuracy of the 
image variable; in particular, given (b pq A b qp ) as a precondition, SI does not falsify this condition, 
because p and q have clock values differing by at most one in the precondition. Therefore it suffices 
to verify that any change to clockp or clock 9 also satisfies the lemma. In a reset-free computation, 
only S3 and S6 change a clock variable. If S3 executes, incrementing clockp, we have clock p < x. p [q] 
as a precondition. Since b pq , we have clockp e {clock g , clockp + 1} also as a precondition; thus the 
increment to clockp results in a state satisfying |clock p — clockp < 2, verifying b pq . The postcondition 
also satisfies b qp , since the change to clockp does not alter the relation between clock g and x q [p\. A 
similar argument applies to S6, and also to the case of q incrementing its clock. ■ 



Proof of Lemma g: By definition of a rising computation, each process has a lower bound on neigh- 
boring clock variables in its x variable, because clock values cannot decrease in a reset-free computation. 
Suppose p is the first of (p, q) to increment its clock. A precondition for this step is clockp < x p [q], 
which implies clockp < clock g , which in turn implies x 9 [p] < clockp. Consider two cases for this last 
inequality, (i) x q [p] < clock 9 or (ii) x q [p] = clock g . For (i), process q cannot increment clock 9 , and 
this situation will persist until p increments its clock sufficiently many times so that clockp > clock g . 
It is straightforward to verify that p does not increment clockp beyond clock g + 1, so for case (i) the 
first increment to clock g establishes (b pq A b qp ). For (ii), we deduce from the inequalities above 
(clockg < clockp A clockp < clock g ) holds as precondition to p's first increment step, and the in- 
equalities with regard to the x variables are similar. So for case (ii), (b pq A b qp ) holds directly. 



Proof of Lemma By Lemma g| neighboring (p, q) establish (b pq A b qp ) at or before a; by Lemma 
such processes continue to satisfy this property for the remainder of the reset-free computation 
segment. ■ 
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Proof of Lemma ^: Because we consider a based computation, and not a rising computation in this 
lemma, the invariance of (b pq A b qp ) stated in Lemma |l| is not applicable. Note that (Vp :: ~^gap p ) 
holds at a smooth state. The invariance of smoothness is therefore verified from the conditions of 
S3-S6, since no gap exists at a smooth state and S3 preserves smoothness. It is also simple to verify 
that the least clock value, if smaller than 7", increments within two rounds from a smooth state, hence 
at most 2T = 0(2?) rounds are needed to obtain a state satisfying (Vp :: clock p = T). A similar 
argument shows that all w variables converge to 32? + 1 within 0(2?) rounds. ■ 

Proof of Lemma ||: In the first round, p reads neighboring clock values and detects local minimality. 
If p increments in this round, the lemma holds; and if p does not increment, it writes its clock and 
detects cEcho in the next round, and local minimality implies p will increment clock p either by S3 or 
S6. ■ 



Proof of Lemma g: Observe that (Vp :: w p = 32? + 1) holds at least until some clock exceeds T — 2? 
so that S4 can execute. And w p = 3D + 1 => wBig p , so process p does not execute the assignment of 
S5. This implies that the computation is reset-free until some clock obtains the value exceeding T — V. 
Lemma [s| implies that each minimal clock value increments in any pair of rounds, which implies that 
the maximum of the set of clock values eventually grows as the computation proceeds. Let clock p 
be the first clock to attain the value 82? + 1 at state a. Thus (Vq : q G JV P : clockp > 82?) holds 
prior to a. More generally, it follows by induction that (Vq : dist 9P = k > : clock, > 82? — k). 
Therefore, each clock value has incremented at least once prior to a. Let clock, be the first clock 
to attain the value 92? + 1 at state /3. At state /3, each process has incremented its clock twice in a 
reset-free computation, implying that each process has read all of its registers at least once in this 
reset-free computation. Therefore the computation segment beginning with (3 is by definition a rising 
computation segment (at least until some clock exceeds T — 2?). Now let clock r be the first clock to 
attain the value 102? + 1 at state 7. At state 7, each process has incremented its clock at least once 
in a rising computation, and by Lemma ||, 7 is a smooth state. ■ 

Proof of Lemma fj]: Lemma ^ shows that the computation contains a smooth state, so the obligation 
here is to show the 0(2?) time bound. By Lemma |5| each minimal clock value increments at least 
once in any two consecutive rounds, so within 202? + 2 rounds, some clock attains the value 102? + 1, 
establishing a smooth state. ■ 

Proof of Lemma ^: The proof begins with a claim on the first k rounds of the based computation: 
within k rounds there is a state satisfying 

(1) (Vg : dist qr = k : w, < 2fc A clock, < 2k) 

(2) (Vg, j : j < k A dist qr = j : w, < 3k — j A clock, < 3k — j) 

The claim is shown by induction. The first state of the computation satisfies the claim for k = as the 
base case. Suppose the claim holds for k < £ and consider two processes q and s such that dist rq = £, 
s 6 J\f q , and dist rs =1+1. Let a be a state satisfying (|l|)-(^) for k — i. By the cEcho condition 
of S3, process q does not increment clock, beyond 21 until process s's image fields in Register^ have 
the appropriate values. In fact, these register fields may initially have the appropriate values, which 
would allow q to increment clock and w variables to 2£ + 1 by S2-S3. However process q cannot 
subsequently increment to 2£ + 2 until the cEcho condition holds, which requires a cycle by s (and all 
other neighbors). Process s therefore observes y s [q] < 2£ + 1 in its cycle and assigns at most 2£ + 2 
to its w and clock variables. Since a occurs at least by round £, the bound of 2£ + 2 for s variables 
applies within round I + 1, which establishes (fy) of the claim. 
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Condition fl2) is also shown by induction. For k = 0, the base case, (g) holds vacuously. Now suppose 
holds for k < £ and consider two processes q and s such that dist rq = £, s E Af q , and dist rs =1+1. 
Condition (Q) places an upper bound on variables at distance £ + 1 from process r within round £ + 1 . 
Therefore clock s < 2(1 + 1) within round £ + 1. In moving from round £ to £ + 1, we consider the 
possibilities for process q and clockg. If clockg and clocks differ by more than one and process q executes 
a cycle, then S5 resets clockg ; before any further change to clockg occurs, the cEcho condition requires 
a full cycle by s, which validates @ up to distance £ + 1 within round I + 1. If clock, and clocks 
are equal or differ by one, then clockg could increment. Observe here that no clock or w variable can 
increment beyond one more than any neighboring value; by another inductive argument, no clock or 
w variable increments beyond j more than any corresponding variable at distance j. Therefore clockg 
does not increment beyond (21 + 2) + 1 so long as clocks < 2£ + 2. This observation is generalized by 
for k = £ + 1 within round I + 1. Note that we have assumed that any clock increment is due to 
S3 and not S6 in this argument; this assumption is justified by (|IJ), since w < 32? + 1, which disables 
execution of S6. ■ 



Proof of Lemma g: Let a be a state satisfying (Vg :: clockg < 32?). By Lemma g such a state a 
occurs with 25 rounds of the based computation. So long as every clock is at most T — T>, no step 
subsequent to a decreases a w variable; and if no w variable is reset by S4 in a consecutive pair of 
rounds, then the minimum value of the set of w variables either increases by that pair of rounds or all 
w variables already have the maximum 325+1 value (we consider a consecutive pair of rounds to ensure 
that wEcho will hold for S2). Therefore, if no clock variable attains the value 72?+ 1 within 2 • (325+1) 
rounds, all w variables equal 325+1 and the lemma holds. On the other hand, if some clock does attain 
the value 725 + 1, we shall deduce that all w values equal 325 + 1, which also proves the lemma. The 
argument rests on the following claim: at all states subsequent to a satisfying (Vp :: clock p < 725), 
the implication clock p > 325 + A: =>■ w p > k holds for every p and < k < 325 + 1. This claim is verified 
by induction on k. For k = the result is immediate from the domain of w variables. Now consider 
k > and suppose the claim holds for k — 1. Let q be the first process to assign clockg <— 325 + k. 
If the assignment occurs by S6 then w — 325 + 1 and the claim holds; if the assignment occurs by 
S3, then each neighbor of q has a clock value of 325 + (k — 1), hence by hypothesis each neighboring 
w variable is at least k — 1, and w q > k — 1 by the same hypothesis. The result is that the same 
cycle assigning clockg <— 325 + k also assigns w q to be at least k. Similar arguments treat the general 
case for q (not necessarily the first) assigning 325 + k to clockp, verifying that w q > k as a result. To 
complete the lemma, consider the first state S where some clock g has value 725 + 1. By the induction 
argument given in the proof of Lemma 0, any clock at distance j from clock 9 has had a value of at 
least 725 — j prior to state S. Therefore every clock has contained a value of at least 625 + 1 prior to 
<5, implying that each w variable is at least 325 + 1 prior to S. The state immediately preceding S thus 
satisfies proof obligation. ■ 



Proof of Lemma 11: by induction on t. For t ~ let a' = a to satisfy the base case. For t > 0, 
we have clock g < it A w q < 3t by hypothesis. By the Echo conditions of S2, S3 and S6, the clock 
and w values of q remain at most 3t until all neighbors either (i) complete cycles that observe these 
values and write corresponding images to output registers or (ii) happen to have these values already 
in their output registers. 

Considering (i), for r G M q satisfying dist pr = t+1, the execution of S2 assures w r < 3t + 1 within one 
round, and clock r is at most 3i + 1 if r observes no gap, or assigned some value at most w r otherwise; 
either case verifies the inductive hypothesis for t+1. These considerations for (i) also verify the 
second part of the lemma, which concerns a path of unperturbed processes, and the same hypothesis 
with 3t replaced by t. 



19 



Considering (ii), process q may increment clock g and w g because r G N q happens already to have 
values corresponding to clock 9 and w q in its output register fields. In this case, q may increment its 
variables to at most 3£ + 1 immediately. Furthermore process r may initially have its program counter 
at S7, about to write its image variables in such a way that q can observe the cEcho condition (even 
though r would not actually read and write in a full cycle). Therefore, if r executes S7, process q can 
increment variables again to at most 3t + 2. However, here a cEcho condition will not be satisfied 
at q until all neighbors complete full cycles, so q's variables cannot exceed 3t + 2 until r completes a 
cycle. When r does complete a cycle, by the reasoning above for (i) we deduce that clock r < 3t + 3 
and w r < 3t + 3 for r G J\f q . ■ 



Proof of Lemma 12: Note that the lemma holds trivially if the initial state is n-perturbed. For the 
case k < n we use induction on t and nested induction on k and suppose a based computation. For 
the base case t = consider p G Af q ■ Since q is perturbed, there is a path P from p to some perturbed 
r (possibly through q) of k + 2 or fewer processes, which is not smooth. Because clock p = T, some 
neighboring pair of processes along path P has the property that one clock exceeds T — T> while the 
other is less than T — T>. Therefore some process in path P executes S4 in the first round. By the 
arguments of Lemma [H] it follows that p executes S4 within k + 2 rounds. This completes the base 
case, but reasoning similar to the nested induction also applies for t > 0. Finally, because the initial 
state may not justify a based computation, two additional rounds are added to conclude a k + t + 4 
bound. ■ 



Proof of Lemma 13: Lemma [h] states that a process executes S4 at most once in a computation, 
so it suffices to show that p either does not execute S4 or executes S4 within the first 4 + minfD, t + k) 
rounds. If p is unperturbed, Lemma |l2| implies the result. If p is perturbed, then for some perturbed 
region P containing p, there is an unperturbed q neighboring some process of P that executes S4 within 
the first 4 + min("D, k) rounds by Lemma [jj]. Applying Lemma [Tl] we deduce that clock p < 3 min(2?, k) 
holds after min(D, k) additional rounds, and by arguments of Lemmas || and ^ process p does not 
execute S4 in the remainder of the computation. Therefore, for perturbed p, the distance from p to a 
perturbed process is t = and after 4 + min(2?, k) + min(2?, k) rounds, process p does not execute S4. 



Proof of Lemma 14: by induction on t. The base case t — trivially follows from the domain of w 
variables, which have non-negative values. The same observation concerning the domain of w variables 
simplifies the proof obligation to the case dist pq < t. It is useful also to observe base cases for t = 1 
and t = 2, since by the end of round two the computation is based, which simplifies reasoning for 
higher rounds. For t = 1 the verification is again trivial by the domain of w variables. For t = 2, it is 
required to show that by the end of round two, w p > 1. In fact any change to w p is an increase from 
its original value of zero, and at least one increment occurs because wEcho p is observed by p within 
two rounds following a. No subsequent reduction to w p results in a value less than one, since wMin p 
is at least zero at all states. This verifies the base case for t = 2. 

Now suppose the hypothesis w g > [(t — dist pq )/2\ for every q such that dist pq < t at some state a'. 
Note that no such process q subsequently executes S4 in the computation, by Lemma therefore 
any subsequent change to w q occurs by S2. If S2 assigns w g a value at least [((t + 1) — dist pq )/2\ in the 
round following a' , or if w q already has such a value and does not decrease, then the induction step 
is verified. Therefore we consider the possibility that w q either remains unchanged or decreases below 
[(t — dist pq ) /2J by execution of S2. A decrease only occurs if w 9 > wMin q + 1, so a decrease below 
[(t — distpq) /2j is only possible if there is a neighbor r G Af q satisfying y [r] < [(t — dist pq )/2\ — 2, 
which would in turn imply that such a value existed in w r in the previous round. But by hypothesis, 
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w r > [(t — dist pr )/2\ , and since r S M q the value of w r is at least [(t — distp q ± 1)/2J , which contradicts 
y g H < L(* — dist pq )/2\ — 2. Therefore such a decrease to w g cannot occur. 

The remaining case to consider is that \N q = \_(t— dist pq )/2\ and does not change in the round following 
a'. Here there are two cases for t and g, either (t — dist pq ) is even or it is odd. If (t — dist pq ) is even, 
then l(t — dist pq ) /2J is equal to |_((f + 1) — dist pq )/2\ and the hypothesis for (t + 1) is proved — the 
value of w g can remain unchanged in the round following a' and satisfy the hypothesis. If, however, 
(t — dist pq ) is odd, then w 9 is required to increment to verify the hypothesis for (t + 1). Observe 
that if (t — distpq) is odd, then [(t — dist pq )/2\ is equal to [((t — 1) — dist pq )/2\, so we infer that 
w q = l(t — distpq) /2J held at round (t — 1) (here we assume the hypothesis not only for t, but (t — 1) 
as well, which is permissible because base cases for t = 1 and t — 2 have been verified). Therefore by 
round (t + 1), process q observes wEcho q and increments w 9 , which verifies the hypothesis for (t + 1). 



Proof of Lemma 15: by induction on t, for t > 0. Note that round t + 2 occurs in a based com- 
putation, since within two rounds following a the computation is based. The base case for induction 
is shown for t — and t = 1 , since the main induction step relies on two previous rounds of a based 
computation. For t < 1, since every clock variable is at least zero, the base cases are verified directly 
by the domain of clock variables — which are at least zero at any state. 

Note that for any t, t — 2 < dist pq trivially satisfies the conclusion because clock variables are always 
at least zero; therefore in the remainder of the proof we consider only the case of q and t satisfying 
t — 2 > distpq. Now suppose the hypothesis holds for t — 1 and t — 2, t > 2, aiming to show that the 
hypothesis also holds for t, that is, that clock variables satisfy the specified lower bound by the end 
of round t + 2. 

By Lemmas [ll] and 12, by round t, any process in the set R = { r \ dist pr < t — 3}. has either 
executed S4 or will not do so throughout the remainder of the computation. Therefore in round 
t + 2, any reduction to clock,, for r 6 R could only occur by S5. Lemma [ll] establishes that w r > 
l((t + 1) — dist pr )/2\ holds invariantly following round t + 1. So if process r executes S5, the result 
satisfies clock r > l((t — 2) — dist pr )/2\, which would verify the inductive hypothesis for r and round 
t + 2. If r does not execute S5 in round x + 1, then consider two cases for r. 

Case: t — dist pr is even. Observe that [((t — 2) — distp r )/2\ differs from [((t — 3) — distp r )/2\ , meaning 
that the obligation is to show that clock r is either at least [((t — 2) — dist pr )/2\ by the end of round 
t + 1, or that clock r increments during round t + 2. If the former holds, the hypothesis is proved, 
so suppose clock,. = |_((£ — 3) — dist pr )/2\ at the end of round t + 1. Because t — dist pr is even, 
clock r > — 4) — distp r ) /2\ by hypothesis for t — 2. But this implies that during round t + 1, 
the value of clock r either did not change or was reduced by S5. However a reduction by S5 would 
satisfy the hypothesis for t as well, because of Lemma |l4|'s bound on w variables. The only remaining 
possibility is that clock,, does not change in round t + 1, implying that r observes cEcho during round 
t + 2. Therefore, if clock r < cMin r when r observes cEcho, then clock r will increment either by S3 
or S6. To show that r does indeed observe cEcho, we use the hypothesis for t — 1 and each q G Af r . 
If distpq < distp r , then by round t + 1 (and throughout round t + 2) the relation clock 9 > clock,, 
holds at least until r increments its clock. If dist pq = dist pr + 1, then l((t — 2) — dist pr )/2\ and 
2) — distpq) /2J are equal, and again the relation clock g > clock,, holds at least until r increments 
its clock. 

Case: t — dist pr is odd. A similar detailed argument can be given for this case, but there is a simpler 
approach: [((t — 2) — dist pr )/2\ and [((t — 3) — dist pr )/2\ are equal, so the hypothesis for t — 1 and r 
directly suffice to verify the hypothesis for t. ■ 
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Proof of Lemma 16: In any computation, cither some process executes a double-reset or no process 
does so. In the latter case, the core system component stabilizes within Ai rounds, and the repair timer 
concurrently reaches the timer-final condition within 0(T) rounds by Theorem [l]. This demonstrates 
M + 0(T) stabilization time if no double- reset occurs; the same argument applies to the case where 
any double-reset occurs by S4 and not by the core system. Lemma |l0| implies that a double-reset 
occurs at most once for each process in this case. 

Now consider the possibility that the core system executes a double-reset at least once in a computa- 
tion. All such assignments cease after the base system stabilizes, which occurs within Ai rounds, so 
the system stabilization time is Ai + 0(T). To show that any process executes a double- reset at most 
once, we demonstrate that the core system stabilizes before clock > T — 2? holds at any process, since 
Requirement || prevents repeated resets of the clock so long as clock < T — V. 

If any double-reset assignment occurs, then within 2? rounds thereafter, a state tr occurs such that 
each clock is at most 32? by Lemma O, and also within 2? rounds, time-accuracy holds and is invariant 
thereafter by Theorem ^|. Although Theorem || is conditioned on k < n for a fc-perturbed initial state, 
its proof arguments are valid for the case of an n-faulty initial state, provided some process executes a 
double-reset in the first round. While we do not suppose that a double-reset occurs in the first round, 
the state preceding the first double-reset can be considered as the initial state for the subsequent 
computation, so that Theorem |^'s results apply for the suffix computation. Time accuracy for the 
extreme case of an n-faulty initial state implies for clock = t that at least (t — T> — 52?) = t — 62? 
rounds have transpired. Therefore, if T — 2? > X + 62?, where X is the number of rounds needed 
for stabilization, then as soon as time accuracy holds, no clock increases beyond T — T> until the core 
system has stabilized. Requirement ^| implies stabilization within T — 72? rounds, which ensures that 
the core system stabilizes before there is the possibility of a second double- reset. To complete the 
proof we address the period between the first double- reset and before time accuracy holds. This is at 
most 2? rounds, and it is easy to show that no clock increases from zero to beyond T — 2? within 2? 
rounds, so a second double-reset does not occur in the period before time accuracy holds. ■ 
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